Model-Free Preference-Based Reinforcement Learning

Specifying a numeric reward function for reinforcement learning typically requires a lot of hand-tuning from a human expert. In contrast, preference-based reinforcement learning (PBRL) utilizes only pairwise comparisons between trajectories as a feedback signal, which are often more intuitive to specify. Currently available approaches to PBRL for control problems with continuous state/action spaces require a known or estimated model, which is often not available and hard to learn. In this paper, we integrate preference-based estimation of the reward function into a model-free reinforcement learning (RL) algorithm, resulting in a model-free PBRL algorithm. Our new algorithm is based on Relative Entropy Policy Search (REPS), enabling us to utilize stochastic policies and to directly control the greediness of the policy update. REPS decreases exploration of the policy slowly by limiting the relative entropy of the policy update, which ensures that the algorithm is provided with a versatile set of trajectories, and consequently with informative preferences. The preferencebased estimation is computed using a sample-based Bayesian method, which can also estimate the uncertainty of the utility. Additionally, we also compare to a linear solvable approximation, based on inverse RL. We show that both approaches perform favourably to the current state-of-the-art. The overall result is an algorithm that can learn non-parametric continuous action policies from a small number of preferences. Introduction One major limitation of reinforcement learning is that a numeric reward function needs to be specified by the user. This is particularly true for complex control tasks as they occur in robotics where the reward function often consists of several hand-tuned terms. Hence, in recent years, the community has worked on rendering reinforcement learning algorithms more applicable by avoiding a hand-coded definition of the reward function. One of these approaches is preference-based reinforcement learning (PBRL). PBRL uses only pairwise preferences over policies, trajectories, states or actions (Akrour, Schoenauer, and Sebag 2012; Wirth and Frnkranz 2013b; Wilson, Fern, and Tadepalli 2012). Many PBRL approaches rely on a model of the system dynamics (Akrour et al. 2014), but often, accurate models are not available and are also hard to learn. Hence, a Copyright c © 2016, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. model-free approach is desirable. Additionally, directed exploration of the utility function is used often, which is hard to perform if the model is unknown or the state-action space is continuous and possibly high-dimensional. In this paper, we show an algorithm for learning a continuous action policy without requiring knowledge of the model or maintaining an explicit approximation of it. The preferences are used to estimate the expert’s utility function. In contrast to traditional PBRL methods, our method is able to use data from interactions with the environment in an online fashion to improve the policy as well as the estimate of the utility function. It can, nevertheless, achieve better performance. The utility function is learned using a Bayesian approach, estimating the uncertainty. An approach that could be used for exploration. Additionally, we compare with a computationally less expensive linear approximation based on ideas of Ng and Russell (2000).